Thanks for the PR! I'll review it properly later.

The first reaction was that we won't need a custom index (to avoid duplication), but you might be right that it's the easiest way.

Questions for you:

• do you keep your document chunks or use the Pinecone to maintain them? (I think so)
• do you embed your documents with OpenAI or use Pinecone?
• do you rerank documents inside Pinecone (I think not?)

(some notes for me)
I'll check for the following functionality

• upload documents + upload embeddings (index composition/some postprocessing of ChunkIndex?)
• getting embeddings (get_embeddings) if we use the embedding by Pinecone
• find closest docs (find_closest if we only return chunk positions, maybe we could hack documents retrieved in a special CandidateChunks)
• getting document chunks (either chunks of subindex or maybe use getindex())

do you keep your document chunks or use the Pinecone to maintain them?

do you embed your documents with OpenAI or use Pinecone?

The chunks and embeddings get uploaded to Pinecone and then Pinecone takes care of sorting/clustering and all that when retrieving.

do you rerank documents inside Pinecone

Well, no, but it's possible I think: https://docs.pinecone.io/guides/inference/rerank

Apologies for scattering comments all over!

One thing, Pinecone should be relevant for "retrieve" step. It should need very little changed for generate! step (only build_context needs work because we build overlap/window extension).
I think it would be good to step back and decide on what do you want to be handled by PT.

Eg, If you say:

• we want to chunk documents externally (not sure why)
• we don't want to build index in PT (ie, chunk, embed, upload)
• we DO want to manage embeddings by PT (risking inconsistency/bad results otherwise)
• we DON't want to use any tag/tag filtering (predicate pushdown)
• we DON't want reranking

Option 1) The the simplest path should be to have a lightweight PineconeIndex + method for retrieve (manually writing everything) + method for build_context (to avoid creating the overlaps which are not possible with 1:1 retrieval).

Option 2) (this also promotes more re-use and optional addition of other methods as the RAG paradigm progresses)
However, the retrieve would be so butchered that it might be better to:

• define No-op for get_tags, find_tags (we have that so just extend), rerank
• add methods for get_embeddings, find_closest (-> uses Pinecone.query), build_context

Option 3) (I would recommend this)
I'd keep the current paradigm: the index (stores chunks + embedding/ remotely) + candidatechunks (stores query results/or its reference).
The difference would be that we create a custom candidatechunks object that will hold the chunks, metadata, sources (not the "references" to index)
We create support for chunks(index,candidatechunks) or we continue the getindex with index[candidatechunks,:chunks].
This is slightly more work/thinking upfront, but then you can re-use all existing methods for embeddings, reranking, build_context, etc. the benefit is that to improve your RAG, you can just try different methods without having to "tweak" each one of them first.

The last option requires also some methods for find_closest (your "query"). And a few extensions to support the No-Op with new Index (eg, in find_tags).

What do you think?

Sorry for butchering your API! I didn't mean to actually hit click on creating this PR, it is definitely far from "ready"!

Thanks for all the comments. I would go for your recommendation (option 3), but I am still unsure how to go about that. @pankgeorg what do you say?

I integrated all feedback, I hope. There's still some TODOs left and some choices left to make, as we discussed. I'll try to include everything below:

Comments

• The view should be properly implemented. Probably this will be done by adding a new struct, SubManagedIndex, similar to SubChunkIndex.
• Names are still a bit weird, I just added them like you first suggested @svilupp cause I couldn't find something better (AbstractCandidateWithChunks).
• Right now, build_index is just calling the PineconeIndex constructor. This could be an option for creating an Index (just calling the constructor); another option could be handling chunking and embedding as well, and adding them to Pinecone. Maybe both would be best?

TODOs:

• Add chunking, embedding etc in build_index
• Add SubManagedIndex + view implementation
• Write docs properly
• Add examples in the docs
• Clean up unnecessary methods
• Clean up dependencies

I got to the cleaning up point. There are also a bunch of TODOs scattered around, minor aspects that need fixing. I will do the end-to-end example once those are finished.

@svilupp would you mind having another look at this, please?

I integrated your feedback @svilupp. I hope I understood your comments well enough. I left some TODOs and also added a MWE for airag.

Thank you for taking it thus far, Iulia!

I hope to take it over and merge it, but I'm short on time these days, so it might take a while before I get to it.

Trying to finish this up but I am facing issues with the Pinecone.jl package. The last released version was v1.0. However, the version that gets installed is v2.0.0. In this later version, there is no longer PineconeContextv3 but simply PineconeContext. There are probably more breaking changes.

Am I missing something? Where is this version 2.0.0 coming from?
@iuliadmtru